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Test generation and test data selection are difficult tasks for model based testing. Tests for a program 
can be meld to a test suite. A lot of research is done to quantify the quality and improve a test suite. 
Code coverage metrics estimate the quality of a test suite. This quality is fine, if the code coverage 
value is high or 100%. Unfortunately it might be impossible to achieve 100% code coverage because 
of dead code for example. There is a gap between the feasible and theoretical maximal possible code 
coverage value. Our review of the research indicates, none of current research is concerned with 
exact gap computation. This paper presents a framework to compute such gaps exactly in an ISO-C 
compatible semantic and similar languages. We describe an efficient approximation of the gap in all 
the other cases. Thus, a tester can decide if more tests might be able or necessary to achieve better 
coverage. 

1 Introduction 

Tests are used in model based testing to identify software defects. High quality test generation and test 
data selection can be difficult tasks when the test has to satisfy a lot of requirements or cannot be created 
automatically because of the undecidability of the halting problem in Turing powerful languages. Given 
requirements for a test suite (set of tests) are functional or non- functional (e.g. execution times, runtime, 
usage of memory, correctness or a minimum value of a code coverage metric). Code coverage metrics 
quantify the quality of a test suite rather imprecisely and guide testers only. There is a gap between the 
feasible and theoretical maximal possible code coverage value. Sometimes demanded requirements are 
unsatisfiable because of gaps. Unnecessary additional tests will be computed while not all requirements 
are satisfied. This enlarges the test suite and introduces redundancy. Fortunately these problems (caused 
by metric imprecisions) can be solved by computing these gaps, which is not possible for Turing powerful 
languages in general. Therefore this paper presents suitable models in a new C-like syntax. These models 
allow to use an ISO-C compatible semantic. In this paper we show how to compute such gaps exactly 
for these models using formal verification techniques resp. software model checking ideas. The paper is 
organized as follows: at first we clarify basics and used notations; we then present our framework, apply 
it to some common coverage metrics and illustrate this on some examples. Finally we discuss related 
work and present a summary and conclusions. 

2 Basics 

2.1 Code Coverage Metrics 7 

Let Tp = 2'^^"^ be the set of all possible sets of tests and P a program written in a common programming 
language such as C, C-i-i- or Java. Each t = {ai , ...} G Tp is a test suite with tests a, for program P. 
The function :Tp ^ [0, 1] is a code coverage metric, if is monotonically increasing. The program 
P can be omitted, if it is well-defined by the context. 
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In this paper some common code coverage metrics for functions, statements, decisions, branches and 
conditions will be considered as examples. Other ones (e.g. linear code sequence and jump coverage, 
jj-path coverage, path coverage, entry /exit coverage or loop coverage) can be adapted in a similar way. 

The function coverage metric )j (0 := \func{t)\/\func{P)\ is the ratio of functions func{t) that has 
been called in the test suite t, to all functions func{P) in P ||T2| . 

The statement coverage metric := \stats{t)\/\stats{P)\ is the ratio of statements rfafs(f) that has 
been executed in the test suite t, to all statements stats{P) in P [ 12|. To distinguish the same statement 
s on different program points l\ and h, we annotate each statement s with unique labels h and h from 
program P, so that l\: s ^ stats{P) and h'-sG stats{P). 
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Figure 1: SPDS example P\ in ISO-C syntax and corresponding BBICFG 

Let blocks{P) be all basic blocks IH in P. Every program point has a surrounding basic block. The 
basic block inter-procedural control flow graph BBICFGp = {blocks {P),edges{P)) (see Fig. [T]l consists 
of the basic blocks blocks{P) as nodes and edges edges{P) C blocks{P)^, where {b\,b2) G edges{P) iff 
there is an execution path of length 1 from the end of block bi to the entry of block b2 (execution of the 
last statement of block bi). The decision coverage metric := \edges{t)\/\edges{P)\ is the ratio of 
executed edges edges{t) of the control flow graph BBICFGp for t, to all edges edges{P) in P. 

The branch coverage metric )f (?) := \blocks{t)\/\blocks{P)\ is the ratio of basic code blocks blocks {t) 
executed during test suite t, to all basic blocks blocks{P) in P [13|. Even if all basic blocks are covered 
by test suite t and Y[{t) = 1, there can be uncovered branching edges in the basic block inter-procedural 
control flow graph BBICFGp. Thus Y^{t) < I is possible in this case. 
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Let bExpr{l) be the set of all Boolean sub-expressions on label I of program P and 



BExpr{P) := {{I ,bExpr{l)) • / G labels{P)}. 



(1) 



The condition or predicate coverage metric )f (?) := \exval{t ,P)\/ {2 ■ \BExpr{P)\) is the ratio of eval- 
uations of boolean sub-expressions exval{t,P) C BExpr{P) x {true, false} of the test suite t, to all 
evaluations of boolean sub-expressions in P |[T2l . The relation exval{t,P) describes the evaluations of 
sub-expressions e on label / under test suite t, such that {{I, e), true) G exval{t,P) iff there is a test a ^ t 
where e can be evaluated to true on label / under test a. When boolean operations are not short circuited, 
condition coverage does not necessarily imply decision coverage. 

2.2 Code Coverage Metric Gap 5 

Let y.Tp^ [0, 1] be a code coverage metric for a program P. The code coverage metric gap 5y{P) € [0, 1] 
is the smallest difference between the coverage ratio of a test suite t ^ Tp and the theoretical maximal 



Let 5x{P) denote 5y^ (P), where x G {c,d,s,f,b}. Obviously dead code can cause 5y(P) > 0. If some 
evaluations of boolean (sub-)expressions cannot be realized, 5y{P) > is possible without dead code 
(e.g. condition or decision coverage). In Turing powerful programming languages the gap 5y{P) can not 
be computed in general, because the halting problem is undecidable. In this case the gap 5y{P) can be 
approximated only. We show how to compute the exact gap 5y(P) for an ISO-C compatible semantic by 
adequate modeling. 

2.3 Suitable Models 

A more expressive model describing the program behaviour allows for a more accurate approximation of 
the gap 5y{P). If the model is not Turing powerful and the model behaviour is equivalent to the program 
behaviour, the gap 5y{P) can be exactly computed. Therefore we defined an ISO-C compatible semantic 
using pushdown systems (PDS) |9]. The split of the ISO-C language definition into platform-independent 
semantics and platform-specific semantics has a serious implication for deciding the halting problem of 
ISO-C programs: whether a C program halts or not depends on the platform-specific semantic. Thus, 
even though the halting problem for a C program is decidable for a platform-specific semantic, the halting 
property can become undefined if no specific platform is assumed . Now we present an extension of 
the PDS used in [9| to symbolic pushdown systems (SPDS) using an ISO-C like syntax. SPDS use a 
more compact representation and define the PDS configurations and transitions symbolically. 

A SPDS is a tuple S = {vgbl,func), where vgbl is a finite set of variables (global variables in ISO-C) 
and func is a set of functions (pairwise different names) with an initial function main G func. Each 
variable v has an integer type[^fe/f5(v) G N>i and a fixed length len{v) G N>i. Every variable v is an 
array. A function is a tuple {f,param,vlcl, stats), where {f,param) is a function signature with a unique 
function name / and a finite list of parameter variables param. The set vlcl is a finite set of variables 
(local variables in ISO-C), such that param C vlcl. The body of / is a finite list of statements stats. Each 
statement I : s stats has a unique label / G label s{f), and fst{f) G labels (f) is the label of the first 
statement in the list stats. We use vgbl = vgbl{S),func = func{S), param = param{f),vlcl = vlcl{f) 



value 1: 



8y{P):= inf (1-7(0). 



(2) 



The Boolean constants false and true are represented via and 7^ like ISO-C. 
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and stats = stats{f) respectively, if 5 or / are well-defined by the context. Let denote func{l) the 
function / for which I G labels (f). Further let be 

vars:=vgblU [J vlcl{f) , stats{S) := [J stats{f) and labels (S) := [J labels{f). (3) 

fEfunc fEfimc /G func 

Similar to ISO-C the SPDS variables are used to build expression Expr using constants and operators. 
The priority and associativity are the same as in ISO-C. 

An expression e G Expr can be strictly evaluated to an integer number [[e]]g^ G ZU {_L} using valu- 
ation functions for global and local variables g : vgbl x Z — > Z and Cf : vlcl{f ) xIa ^ The symbol 
_L denotes arithmetic exception (e.g. division-by-zero or index-out-of-bounds). The functions g{v,i) and 
Cf{v,i) return the current value of variable v at index / (value of v[/]). The evaluation functions g and Cf 
can be omitted, if they are well-defined by the context. A variable usage v[/] of variable v G vars with 
index / G Z is evaluated as 



VI 



Iff - 



g{v,i) V G vgbl AO < i < len(y) 
c{v,i) vevlcl{f) AO <i<len{v) 
J_ otherwise. 



(4) 



For e G Expr and a statement I : s £ stats{f ), s has one of the following forms: 

• v[ei] = e2', corresponds to writing the value [[^2]] into the variable v at index [[ei]]. 

• /(vi, . . . ,v„); corresponds to a function call (call by value), iff {f,param) is a signature, where 

param = [pi,p2, ■ ■ ■ ,Pn], Vj G vars and bits{vi) < bits{pi) for all I <i <n. 

• return; corresponds to a function return. 

• if (^) goto I'; corresponds to a conditional jump|^ to label /' G labels {/). 

The exception of a dynamic type mismatch occurs for "v[ei] = e2'" and the system terminates, if [[^2]] = -L 
or the type bits{v) is too small to store the value [[^2]] or [[^i]] ^ {0, 1, . . . ,len(y)}. We denote v = e for 
v[0] = e and v for usages of v[0] to emulate syntactically non-array variables. The system terminates on 
statement "// (e) goto I';" too, if [[e]] = _L. The predefined function rand{e) returns a random number 
between and \e\ for e G Expr, whereby rand{l.) = _L. Further ISO-C statements and variations for 
other languages can be mapped to these basic statements in the modeling phase. All variables (global and 
local) are uninitialized and have initially a random value. A test a for 5 is a subset of global variables 
with predefined values for label fst{main). A configuration s = [(/«,c„), (/„_i,c„_i), . . . , (Zi ,ci)]) 
of S represents a state of the underlying Kripke structure with the current execution label /„ G labels, 
the valuation g : vgbl x Z — )• Z of global variables and the stack content. The stack content consist 
of a list of function calls with current execution labels /,■ G label s{S) and valuations for local variables 
Ci : vlcl{func{li)) x'L^'L. The head of s is head{s) = {g, {ln,Cn))- The set of all possible configurations 
is conf{S). A run of S is a sequence of consecutive configurations beginning with an initial configuration 
{ginit, [fst{main),Ci„it]) G conf{S). SPDS are (like PDS) not Turing powerful and can be used to model 
the behaviour of (embedded) ISO-C programs. There is no restriction on the recursion depth. 



^intra-procedural 
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3 Exact Gap Computation Framework 

Let 7 : Tp — )• [0, 1] be a code coverage metric for a program P. Our framework to compute the gap 5y{P) 
consists of the following steps: 

1. If necessary, create a SPDS model S with ISO-C compatible semantic for program P. 

2. Modify the model 5 to a SPDS model 5' to enable gap analysis for the code coverage metric 7. 

3. Compute exact variable ranges for some new variables in S'. 

4. Conclude the exact size of the gap 5y{S) in S for the code coverage metric 7. 

5. Conclude the size of the gap in P. 

3.1 SPDS Modeling (step 1) 

If the given program P is not written in ISO-C (e.g. Java) or P has another platform-specific semantic, we 
create a SPDS model S for P by abstraction. Otherwise the behaviour of 5 is the same of P by mapping 



all the ISO-C statements to the basic SPDS-statements of section 2.3 using abbreviations (described in 
this section). Java can be handled using the tool JMoped Q. Often other languages and correspond- 
ing statements can be mapped to the basic SPDS-statements in a similar fashion. For simplification we 
present some common mappings, which are abbreviations for previously defined basic SPDS-statements. 
We sketch the ideas only, because of limited space. Fig. [T] shows the SPDS example Pi in ISO-C syn- 
tax, where "char x" in line 1 is an abbreviation for "int x(8)[l]"to declare an integer array of type 
bits{x) = 8 and len{x) = 1. 

Omitted Returns and Labels: If there is no return statement at the end of a function body, its existence 
is assumed during interpretation of the symbolical description of S. The same holds for statements with- 
out labels, such that each statement in the SPDS has a unique label after interpretation. 
Parameter Expressions: Basic SPDS-statements allow variables to be passed as parameters in function 
calls. We can simulate to pass expressions by temporary local variables. Let "/ : f{ei,e2, ■ ■ ■,«„);" be a 
function call with expressions et G Expr, where {f,param) is a signature with param = [pi,p2,...,Pn\- 
We introduce new local SPDS-Variables pej ^ vars with type bits{pei) = bits{pi) and len{pei) = 1. These 
variables are used to evaluate the expressions before the function call: pei = e,. Instead of ei now pei is 
passed to / using the basic SPDS-statement/(pei,pe2, • • • ,P^n)- The function call "/ : f{e\,e2, ■ ■ ■ 
is interpreted as "/ : pei = e]_;pe2 = 62',- ■ -pen = en;f{pei,pe2,---,pen)"- Now only basic SPDS- 
statements are used. The code coverage metrics are adapted accordingly. For example the statements 
pei = et; are ignored for the statement coverage metric. 

Return Values: A function can return a value. This value can be used to set a variable "v = /(...);". If a 
function returns an expression e via "return e", a new global variable retf ^ vars is introduced. The type 
of retf equals the return type of function / and len{retf) = 1. The statement "return e;" is interpreted as 
"retf = e;return;". On the other hand the assignment "v = /(...);" is interpreted as "/(...);v = retf-" to 
store the return value of / in v. 

Function Calls in Expressions: If there is a function call /(..) in an expression e, this function is eval- 
uated in a temporary local variable. Boolean operations in SPDS are strict and not short circuited. Short 
circuited expressions (also increments i++ and decrements i — ) can be mapped to strict expressions 
without side effects by several conditional statements. Thus every function call /(..) in an expression e 
will be definitively evaluated during the evaluation of e. Accordingly it is safe to do every call before 
the evaluation of e. Sometimes the order of this evaluation is implementation defined (as in ISO-C) and 
depends on the source language (e.g. i++*i++). Thus we use priority and associativity for calculating 
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this order. The intermediate representation of a compiler can be used, too, to achieve a mapping to SPDS. 
Unconditional Jump: The statement "goto /;" is mapped to "// (1) goto The dead branching edge 
to the following statement is ignored by the decision coverage etc. 

Skip Statement: Particularly low level languages have often a "no operation" statement. We use the 
statement skip, which does not change variable settings. The statement "/ : skip;" can be interpreted 
using the conditional branch "/ : // (0) goto /;". If there is a global variable v G vars, this can also be 
interpreted as "/ : v = v;". The former needs no consideration for the coverage metrics. 
Random Numbers: In ISO-C the function rand{) returns a pseudo random value between and the 
constant RAND MAX, where RAND MAX depends on the system. We can map this behaviour using 
the SPDS function rand {RAN DM AX). A similar mapping for random numbers is possible in other lan- 
guages. 

Conditional Statements: Let si and s2 be lists of statements. The conditional statement "// (e) 
si else s2;" is interpreted as "// (e) goto li;s2;goto : sl;l' : skip;", where h,l' ^ labels, "if (e) s;" 
is an abbreviation for "// (e) s else skip;". 

Local Variable Definitions: A local variable can be defined during an assignment of a basic block or a 
loop header. Such local variable definitions are mapped to local variables of the surrounding function. 
Renaming can be done easily if necessary. 

Loops: A for-loop of the form "for {init; cond; inc) body;" is interpreted as "init; I : body; inc; if {cond) 
goto I;", where / ^ labels. The do and while loops are interpreted in a similar way. 
Modular Arithmetic and Integer Overflow: The ISO-C standard says that an integer overflow causes 
"undefined behaviour", meaning that compilers conforming to the standard can generate any code: from 
completely ignoring the overflow to aborting the program. Our solution terminating the system is con- 
form to the ISO-C standard. Evaluations of expressions in SPDS are not restricted to arithmetic bounds, 
but dynamic type mismatches are possible for assignments v = e.ln the case of modeling nonterminating 
modular arithmetic the modulo operator % can be used to shrink the expression e to fit the size bits{v). 
Hence, a dynamic type mismatch does not occur. 

Dynamic Memory and Pointers: In ISO-C a certain amount of the heap can be reserved using the 
function malloc{int). It returns an address on the heap. The heap is finite, because the number of ad- 
dresses is finite. This behaviour is simulated using a global array heap of type bits{heap) = 8 with 
length lenQieap) = m and a global variable ptr with type bits{ptr) = \log2{fn)'\ and len{ptr) = 1, which 
points to the next free space in the heap array. The function malloc{int) can be implemented as shown 
in Listing [T] with 1024 heap elements respectively, which needs a 10 bit variable ptr for accessing. A 
memory exceptions occurs (label memout), if there is not enough memory left. 

int heap (8) [1024] ; 
int ptr (10) ; 

int(lO) mallocdnt n(10)) { 

if (ptr >= 1024-n) goto memout; 
ptr = ptr+n; 
return ptr-n; } 

Listing 1: Malloc as SPDS in ISO-C like syntax 

Once reserved space can be reused, because a garbage collector and a function free can be implemented 
in SPDS. A pointer is a SPDS variable used as an index of the heap array and an address is just another 
index (returned by the address operator &). Variables placed in the heap array support the address 
operator in contrast to the other SPDS variables. If putting a local variable of a function / into the heap 
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array, the recursion of / will be bound, because of a finite maximal heap size. Coverage metrics have to 
adapt to these additional SPDS functions, statements and variables to be able to compute the correct gap. 
Call by Reference: Instead of passing a variable as a function parameter, a pointer can be used to 
indirectly access variable values in the heap. Thus call by reference can be simulated. Unfortunately this 
results in bounding the recursion, too. 

Dynamic Arrays: Array semantic in ISO-C is defined by pointers and access to its elements is defined 
by pointer arithmetic. Thus malloc{int) can be used for this purpose. 

Other constructs and statements from other languages (e.g. classes, structs, objects, dynamic param- 
eter lists, etc.) can be mapped in a similar way. If an arithmetic exception occurs, the SPDS ends and the 
corresponding ISO-C program P can have undefined behaviour according to the language specification. 
P can terminate, which is a complying behaviour. Therefore this behaviour is used for our modeling 
process. Other implementation defined behaviour can be modeled similarly. 

3.2 Extraction of Exact Variable Ranges (step 3) 

In step 2 a SPDS S' is created for the SPDS S by slightly modifying S (explained in the next section). For 
a PDS B an automaton Post*{B) can be computed, which accepts all reachable configurations of g[T8]|. 
Thus for the SPDS 5' a similar automaton Post* {S') can be created, because S' is just a symbolical PDS. 
This is a basic step in symbolic model checking using Moped [14]. We use the Post* algorithm of the 
model checker Moped for our implementation by mapping our SPDS definitions to the input language 
Remopl£0 ini. The set of reachable heads := {(g,(/,c)) • {g, [{I, c)... ]) e Post* {S')} h finite 
because of finite variable types. Thus exact variable ranges can be extracted from h{S'). Let v E vars and 
I G labels, then rangef (v) := { [[v]]^ • {g, {l,c)) G h{S')} is the exact variable range of v. The notation 5' 
can be omitted, if 5' is well-defined by the context. For all values k G rangei{v) there is a run of S' , such 
that [[v]] =kon label 1 and vice versa. 

h{S') and rangei{v) can be computed symbolically out of Post*{S') using Ordered Binary Decision 
Diagrams (OBDD) operations. The computation ofh{S') is a straightforward OBDD restriction operation 
in Post*{S') and results in a characteristic function g : {0, 1}" — ?■ {0, 1} represented as an OBDD. The 
input vectors of q are heads h{S') encoded as finite Bit sequences. The computation of rangei{v) uses 
cofactors. A cofactor of q is q[xi = b\{x\,X2, ■ ■ ■ := q{x\,X2, ■ ■ ■ ,Xi-i,b,Xi+i . . . ,Xn) lISl. The positive 
cofactor is q[xi = 1] and the negative cofactor is q[xi = 0]. A characteristic function r : {0, 1}™ — )• {0, 1} 
for rangei{v) can be computed using cofactors: 

Lemma 3.1 Let k be the starting index of the encoding ofv on label 1 in q and let m be the length of the 
encoding. Then r{yi,y2, ■ ■ ■ ,ym) = 1 is valid, iff q[xk = y\\[xk+\ = 3^2] • • • [-^/t+m-i = Jm] is not always 
(not the empty OBDD). 

The proof is a consequence of the definitions. The computation of exact variable ranges is more time- 
consuming than model-checking the reachability in 5' [8]. Fortunately rangei{v) can be approximated 
using static data fiow analyses and test suites. This is the case for focusing on efficiency or unbounded 
recursion depth in combination with unbounded parallelism. For further reference in comparisons, ex- 
planations, and proofs see HSJ. 

3.3 SPDS Supplementation (step 2) and Exact Gap Inference (step 4) 

Now we show exemplary, how to apply our framework to common code coverage metrics, 
^e.g. we map integers to nonnegative numbers, as Remopla does not support negative integers 
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3.3.1 Function Coverage Gap 5/(5) 

We supplement S with a new global variable v ^ vgbl{S) using type bits(y) = 1 and len{v) = 1 without 
any assignment or reading usage on v to ensure the existence of at least one global variable in S' . By 
construction this variable v has a random undefined value [[v]] G {0, 1} on each label / G label {S') resp. 
on each program point. The exact function coverage gap 5f{S) can be concluded from the exact ranges 
of variables in 5" as follows: 

Lemma 3.2 

I {/ G func{S) . rangef^^^f^ (v) / 0} | 

df[S) = 1 — T— (5) 

\func{S)\ 

Proof (sketch): The new global variable v does not influence the model behaviour. All variable eval- 
uations and reachable labels in S' are the same in S. Further it is vgbl{S') = vgbl{S) U {v}, func{S') = 
func{S) and func{S) / because of main G func{S). The main observation is, that a label / G label s{S) 
is unreachable or dead, iff rangef (v) = 0. Thus a function / can be called, iff range^^^^^j^ (v) / 0. Choose 
a test suite t' G Ts such that \func{t') \ is maximal. With the maximality of t' we have 

\func{t')\ > |{/ G func{S) • range%^f^{v) / 0}|. (6) 

Every function / G func{t') has a cover witness test a G t' , so that the label fst{f) is reachable under 
test a. Thus it is range^j^^^^j^ (v) / 0. On the other hand we obtain 

\func{t')\ < |{/ G func{S) • range%^f-^{v) / 0}|, (7) 

because each / G func{S) with range^^^^^^^iv) 7^ has at least one test a' (not necessarily G t') to cover 
the function /, which can be detected by an evaluation of v. Accordingly it is 



sup \funcit)\ = |{/ G func{S) • range f^^y.^{v) 7^ 0}|, (8) 
teTs 



which is equivalent to 



{/ G func{S) . ranged , {v) / 0} 
inf 1-7/0 =1 , r^v^ • (9) 

□ 

The exact branch coverage gap 5^(5') can be computed similarly. Instead of function entry points, 
just the block entry points are considered. 

3.3.2 Statement Coverage Gap 5, (S) 

The SPDS 5' will be supplemented with a new variable v ^ vgbl{S) and the type bits{v) = 1 and len{v) = 
1 similarly to the function coverage gap. The exact statement coverage gap 8s{S) can be computed: 

Lemma 3.3 

5(S) = \ " ^ ^ '^""^'^^^ ' ^^"g^^'(^) ^ '^^l (10) 
* \stats{S)\ 



Proof (sketch): Similar to Lemma 3.2 prove sup^gj^. \stats{t)\ = \{l : s £ stats{S) • rangef (v) 7^ 0}|. 

□ 
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3.3.3 Decision Coverage Gap 5^(5') 

The branch coverage uses the nodes of the control flow graph BBICFGs and the decision coverage uses 
the edges. The execution of an edge (^1,^2) £ edges{S) in the control flow graph BBlCFGp depends on 
several conditions such as arithmetic overflow, division-by-zero or boolean expressions for conditional 
branches. To compute the exact decision coverage gap, we introduce a new global variable v,„ ^ vars{S) 
into 5' with type Z?jY5(v,„) = I + \log2{\blocks{S)\Y\ and/en(v;„) = 1. Each label / belongs to a basic block 
bi, which can be identified by a unique number n^, G N>o. This number is assigned to the variable v,„ to 
detect the past basic block for a statement. The type bits{vi„) is big enough to store every unique identifier 
nhj. Each statement "I : s" e stats{S) is modifiecrjto "1 : v,„ = w^,; /' : s" in S', where 1' ^ labels{S) is 



unique. So it is possible to determine the past basic block on label / using the exact range of the SPDS 
variable v,„ G vgbl{S'). The exact decision coverage gap 5d{S) can be computed: 

Lemma 3.4 

\{{a,b) e edges{S) • Ua G range^' (,Jvin)}\ 

= ' — — 

Proof (sketch): Let a,b G blocks{S) be basic blocks. By construction it is G range^^^^^^^ ivi„), iff there 
is an execution path from the end of basic block a directly to the first label fst{b) of basic block b in 5. 
This is equivalent to the existence of a test a, such that {a,b) G edges{a). Thus we have 

3a et' : {a,b) G edges{a) n,, G range f^^^^i^^iytn) (12) 

for a chosen t' G Ts, where \edges{t') \ is maximal. Hence it is 

sup\edges{t)\ = \{{a,b) G edges{S) • n,, € range^^j(^)(vi„)}|, (13) 



which shows ( [TT] ) similar to Lemma 3/2 



□ 



3.3.4 Condition Coverage Gap 5c{S) 



For the condition coverage all boolean sub-expressions (conditions BExpr{S)) on each label are con- 
sidered. The theoretical maximal value can be achieved, when every condition {l,e) G BExpr{S) can 
be 1 (true) and (false). For each boolean sub-expression b £ B = [jiQiabeis{s)^^^P^O) we intro- 
duce new boolean global variables Vh ^ vars{S) with type bits{vb) = 1 and len{vb) = 1 into S'. Let 
further bExpr{l) = {ei,e2, ■ ■ - en} be the set of all boolean sub-expressions on label /. Each statement 
"Z : s" G stats{S) with bExpr{l) 7^ will be modified to "Z : Vg, = ei; v^^ = e2', • • - Ve^ = 1' : s" in 5", 
where I' ^ labels{S) is unique. The statement "Z : s" G stats{S) will be modified to "Z : skip; I' : s" in 
S' , if bExpr{l) = 0. Hence the existence of label I' G labels{S') is guaranteed. Thus the exact condition 
coverage gap 8c{S) can be computed: 

Lemma 3.5 

i&Msis) \range^i,{ve)\ 

5c{S) = 1 - ^^^^ — — (14) 

^ ' l-\BExpr{S)\ 



Additionally this can be done using the native synchronous parallelism in SPDS without an extra label: '7 : v,„ = n^, , i". 
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Proof (sketch): Choose at' ^Ts such that \exval{t' ,S) \ is maximal. Then it is 

{{l,e),b) £exval{t',S) (15) 

4^ expression e can be evaluated to G {0, 1} on label Z in S (16) 

44> b erangef{ve). (17) 



This proves ( 14 1 similar to Lemma 3.2 because of 



sup\exval{t,S)\ = ^ \rangef (Vf,)]. (18) 

t&Ts lelabels{S) 
e^bExpr{l) 



□ 



3.4 Conclusion for 5y{P) based on 5y{S) (step 5) 



The computed gap 5j{S) is exact {5y{S) = 5y{P)), if the behaviour of S is equivalent to the behaviour 
of P. Hence step 1 does not abstract nor simplify the program behaviour. This is the case for our ISO- 
C compatible semantic 191 on C programs. If S abstracts from the behaviour of P, 8y{S) is only an 
approximation for 5y{P). The approximation degree depends on the degree of this abstraction. 



Gap Approximation using 5y and 5. 



+ 



The gap can be approximated by abstracting the program P to a simpler behavior of SPDS S as shown 
above. On the other hand the exact variable ranges rangei{v) can be approximated, too. This is a more 
practical approach particularly for huge software systems. Let rangef{v) be an over- and rangej{v) 
an under- approximation of rangei{v). The sets rangej{v) can be realized using a test suite t G Tp. All 
occurring variable values during the tests a € f can be used as lower bound for rangei{v). On the 
other hand range^(v) can be realized using a conservative data flow analysis. This usually results in 
additional variable values, which never can be achieved. Both 5y and 5:^ can be defined similar to By 
using rangej{v) and rangef{v) instead of range i{v). It is easy to realize, how to bound the gap 5y using 
rangej (v) and rangef (v) : 

Lemma 4.1 5y < 5y < 5y . 

Obviously the gap approximation is perfect and an exact gap is found, if 5y = 5y. In this case it is not 
necessary to compute exact variable ranges. 

5 Exemplary Illustration 

As a comparative measurement of our method the values calculated by gcov |[22l |^ are presented at the 
end of this section. The free tool gcov calculates the code coverage during an execution, which can be 
used to track the code coverage of a test suite. 

To show the concepts presented so far, we use example Pi in Fig. [T]and example P2 in Fig [2] The 
constructs in the presented ISO-C code are automatically mapped to SPDS-statements as described in 
section 3.1 Pi contains an arithmetic exception, caused by a division-by-zero. Hence Pi contains a 
lot of dead code and any test suite with at least one test would be complete (i.e. there is no way to 
cover more code). There is no test necessary (t = 0), because the variables x and y are initialized on 



http : / / gnu . org/sof twcire/ gcov 
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labels 10 and II. Thus it is rangei{x) = rangei{x)^ = {0} and rangei{y) = rangei{y)^ = {1} for all 
/ G L, where L = {10,11 J2,lb0,lb\}. It is rangei{x) = rangei{x)^ = rangei{y) = rangei{yY = for 
all / G labels{Pi) \ L. All the conditions in conditional branches are considered to be statements (see 
BBICFG in the right part of Fig. [TJ, because a conditional branch can contain a statement (e.g. x=0 in 
"if (x=0)..."). 

Thus it is \stats{Pi)\ = 15, \blocks{Pi)\ = 12, \edges{Pi)\ = 15, BExpr{Pi) = {{Ic0,x==0),{lb0,y < 
x) } and exval {t,Pi) = {{ {lbO,y < x) , false)} . Additionally Pi is supplemented with variables v, v,„ , Vx==o 
and Vy<:x to program P[, where the range values can be computed accordingly. An inter-procedural con- 
servative interval analysis |l20l|2ll can detect range i{v) = range i(y in) = for all I ^L' = {IbO' ,lc2} 
and rangei{v) = {0, 1} for all I G labels{P[) \ L' . This is used to compute 5^ , 5^ and 5+. The edges 
{b2,b3), {bl ,b9),{b9,b2), {b'i,b\0), {b3,b5) G edges{P\) are never executed, which is discovered by the 
interval analysis. This results in 5^ {Pi) = ^. Thus, the coverage metrics and gaps of Table [l] can be 
calculated for Pi as described in the previous sections. The values were obtained using our current imple- 
mentation of the program described in fSl. Computing the values presented in Table[T]takes less than two 
seconds on a modern Core 17 CPU equipped with 8 GiB RAM. Table [T] also contains the approximated 
values as presented in section |4] Although the coverage metrics are far less than 100 %, the test suite t is 
complete. Additional tests can not improve these coverages as confirmed by the gaps. 
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0.13 
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Table 1: Code Coverages and Gaps for Pi in Fig. [T]and P2 in Fig. [2] 



label {P2) 


range^ (x) 


range^ (y) 


range^ (z) 


range^ (w) 


m 1 ,m2,m3,m5 ' ,lcO,lc 1 ,lc2 














m0,m4 


{0,1,10} 


{1,5} 


{0,1,10} 


{1,5} 


m5,m6 


{0} 


{1,5} 


{0,1,10} 


{1,5} 



Table 2: range^ in P2 using test suite t for P2 in Fig. [2| 
The test suite t discovers rangej{v) = {0, 1} for each reachable label / in t. 

Most compilers, i.e. GCC and CL|^ from Microsoft, are not able to do a flow-sensitive, context- 
sensitive inter-procedural analysis needed for a more precise lower bound in this example. The abstract 
interpretations done in a compiler or analysis tool do not yield such a precise lower bound, as most other 
tools are essentially model-checkers. Hence, the lower bound on the range for x and y would include all 
possible values at label Ibl in Pi. Thus the lower bound on the function gap would be 0. 

Additionally, using the tool gcov to compute the coverage of the test suite of example Pi , no coverage 
is achieved by any test suite, because gcov does not take arithmetic exceptions into account resulting in 
0% coverage. Of more practical relevance is the calculation of the coverage gap for non-arithmetic 
errors. For instance example P2 in Fig. [2] has a difficult condition {x <y && z > w). The variables 
X, y, z and v of P2 (Fig. |2]) are global variables of type char = bits{..) = 8. The whole block below 
(ml — m3) becomes dead, if the condition on label mO evaluates to false. Thus commit is not called and 

^Shipped with Microsoft Visual Studio 



54 



Gap Computation for Code Coverage Metrics in ISO-C 



1 


char 


x,y ,z ,w; 


2 


void 


commit () { 


3 


IcO : 


y = X + w; 


4 


Icl : 


main ( ) ; 


5 


lc2 : 


return ; } 


6 








7 


void 


main(){ 


8 


mO 




if(x<y && z>w) { 


9 


ml 




X = X + 1 ; 


10 


ml 




commit ( ) ; 


11 


m3 




X = 127; } 


12 


m4 




x=z *(y+x— y)— x*z ; 


13 


m5 




if(x==127) y = 0; 


14 


m6 




return ; } 



-► ^ mO : if (x<y && z>vY] bl 
trua '"'"--N.^alse 



ml : 


x=x+l 


m2 : 


commit ( ) 


t 


IcO 


y=x+v; 


Icl 


main ( ) ; 




b8| 

m3 :x=12 7; 



m4 
m5 



x=z* (y+x-y) -x*z 
if (x==127) 



true 



b4 



2 




/alse 



b5 



m5' : y=0; j"^ rn6: return; 



lc2 : return; 



Figure 2: SPDS example P2 in ISO-C syntax for dead code by bad condition + corresponding BBICFG 

the (indirect) recursion not started. Additionally, for all possible test cases, the condition (x == 127) on 
label m5 never evaluates to true. 

Let t = {(0, 1,0, 1), (1, 1, 1, 1), (10,5, 10,5)} be a test suite with {x,y,z,w) being the values set be- 
fore calling main. It is \stats{P2)\ = 11, \blocks{P2)\ = 8, \edges{P2)\ = 10, BExpr{P2) = {(mO,x < 
j), (mO,z > w), (mO,x < jSlSlz > w), (m5,x == 127)} and exval{t,P2) = {((mO,x < y),true), ((mO,x < 
y), false), ((mO,z > w),true),{{mO,z. > w), false), {{mO,x <y&&z > w), false), {{m5,x == 121), false)} 
The test a = (0, 1, 1,0) would be a good candidate for the test suite t, because 7i({o;}) = 91% is perfect 
(proofed by the gap 5s). Table [T] contains the calculated coverage metrics and gaps for P2. As one can 
see from the third line of Table [T] the exact gap in the existing code is rather small: it consists of the 
condition x == 127 on label m5 and the following code block. This is one of the examples in which our 
method can instruct the tester to expand the test suite. More code can not be covered, because in each 
test the variables x and z as well as y and v are aliases. Although most of the code in the example is 
alive. As seen in the previous example, the approximated lower and upper bounds are not perfect. An 
upper bound on the gap 5f of called functions, is = 1 — 0.5, whereas from the two available functions 
one was called during the execution of test suite t. It is rangei{v) = {0, 1} for every label / G labels{P2) 
in P2 (supplementation of Pg). P2 is also supplemented with variables Vin,Vx==Q and Vy<^x, so that the 
range values can be computed and approximated using an inter-procedural conservative interval analysis 
(range^). The results for the variables are shown in Table |2] and [T] 

Contrary to gcov the computation of the code coverages followed the C Program and did not rely 
on any symbolic assembler. Such abstractions might cause more coverage shown than the actual cov- 
erage in ISO-C. The statement coverage reported by gcov corresponds to js- Values close or exactly 
corresponding to yx can be obtained from gcov for these particular examples. Not all values will match 
Yx, because gcov uses a different definition for decision and branch coverage and relies on symbolic 
assembler output. 

6 Related Work 

A lot of research is done to get a better coverage for a test suite H. However an important point is often 
missing: often it is impossible to cover 100% of the code in practice, because of gaps. 
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To the best of our knowledge no research has been done to compute provable exact gaps used in code 
coverage. Conservative strategies underestimate the coverage gap f5 |. Current research only approxi- 
mates the gap. For instance [3] presents a method to automatically add tests by computing a gap of code 
covered by the test suite and possible code coverage. The authors miss the important point of having 
code which will and can never be used. In |[3l the emphasis is on large scale projects, but especially in 
such large projects there is code which cannot be executed and should be removed by the compiler. As 
ifm describes, some code parts are more important than others. Testing parts of a program which will 
never be executed is then a loss of resources. Gittens et al. use a domain expert to categorize the code, i.e. 
for which parts of the source code their tool should generate tests automatically. Our gap computation 
presented in this paper could be used to automatically categorize the code and not depend on a domain 
expert. Another project of interest is 1161 by Kicillof et al., which shows how to create checkable models. 
The focus of Kicillof et al. are models which can be created by stakeholders or maybe even marketing 
experts, and thus is directed at their specific problems at Microsoft. Most other research concerning the 
computation of gaps in coverage targets the pre-silicon design validation, i.e. |!6llT9l. 

Both papers on pre-silicon design validation are not concerned with testing gaps. They rather check 
if a specification can be achieved. However, our paper is concerned with languages similar to C, not any 
Register Transfer Language (RTL) or even specifications. 

As Regehr correctly writes in [15], such specifications, which are checked in ||6l[T9l might have been 
wrong in the first place. One solution proposed by Regehr for finding errors in specifications is having 
more people to look over these. A different solution uses our tool and computes parts of the realized 
specification that are never used, thus giving hints to erroneous specifications. 

Whereas Berner et al. are targeting the user of an automatic test system [4l our method targets the 
automatic test system itself. Berner et al. describe lessons learned from their experience with code 
coverage analysis tools and automatic test generation tools and propose a list of rules to be followed 
when introducing and using an automatic test tool. Our research was not concerned with usability and 
group dynamics in a programming environment. 

To the best of our knowledge the current research in testing, be it concolicj^or model-based, is not 
concerned with the actual problems of code coverage gaps. Gap coverage analysis is not only useful 
in test case generation but also in the verification of functional correctness. Imagine the case of a dead 
function granting more user rights, it is easy to use a buffer overflow to trigger this functionality. Similar 
methods have been used by the CCC for analyzing and using a trojan horse 1 10 

Another important tool, which might be able to compete with our method is Frama-C |[5]|^ Frama- 
C is a conservative analysis tool, which is able to find dead code, execute a static value analysis and, 
contrary to gcov, is able to detect runtime-errors triggered for instance by division-by-zero. One of the 
differences between Frama-C and the method we propose in this paper is the theory behind it. In contrast 
to Frama-C |,5J our method uses exact computation, does not overapproximate the values and does not 
rely on an experienced user. Our exact value analysis produces neither false negatives nor false positives 
as in Frama-C. Although their value analysis sometimes detects that a function does not terminate, it 
cannot be used to prove that a function terminates in general. 

Frama-C provides sophisticated plugins, but not all of them handle recursion properly. No sophis- 
ticated examples can be handled by Frama-C's value analysis. Some of the examples tested even cause 
runtime-errors in Frama-C itself, thus it is not reliablj^ 

^interwoven concrete and symbolic execution 
^especially the section Upload- and Execute-Mechanismus 
^http : //f rama-c. com 

"^It should be noted, that these runtime-errors should vanish in future versions 
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As our review of the research indicates, none of current research done in testing is concerned with 
exact gap computation. 

7 Summary and Conclusions 

This paper presents a framework to compute exact gaps between the feasible and theoretical maximal 
possible code coverage value. For specifying programs in an ISO-C semantic we use a very powerful 
model, namely SPDS. The power of SPDS allows to model an ISO-C compatible semantics for programs 
without abstraction. Therefore we are able to do an exact value analysis using model checking techniques 
and so we obtain exact gaps. We describe how to efficiently approximate the gap in all the other cases. 
When using flow-sensitive, path-sensitive, inter-procedural and context-sensitive data flow analyses for 
approximating the exact values one can also use a model-checking tool. The biggest problems of using 
a model-checker are false positives or false negatives caused by abstraction. Instead, our approach does 
not rely on such heavy abstraction and does not cause false alarms on our ISO-C compatible semantic. 
Thus user input or feedback is not required to decide about false alarms. A lot of computing power is 
required for using such powerful models. Due to smaller programs and smaller data types our approach 
is still practical for embedded systems. 

Having combined the best parts of model-checking and static analyses we use expansive model- 
checking only when needed (e.g. the gap approximation bounds are not small enough). Thus the compu- 
tation of Post* is needed only if the gap approximation using static analysis and a test suite is not exact 
(5-/5+). 

Using our method a lot of metrics can be compared better among each other now, because of exactly 
specified gaps. Our method allows the testing of non-functional requirements, too. For example the 
worst case execution times (WCET) using a WCET metricf] can be computed. 

Our current research considers the practical relevance of exact gap computation for verification of 
software especially in the area of compiler correctness. Additionally we are considering other areas of 
research to apply the computation of exact values and exact gaps. For example the computation of exact 
value ranges can be used for verification of components L2il . 
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